Speech recognition using electronic device and server

ABSTRACT

An electronic device is provided. The electronic device includes a processor configured to perform automatic speech recognition (ASR) on a speech input by using a speech recognition model that is stored in a memory and a communication module configured to provide the speech input to a server and receive a speech instruction, which corresponds to the speech input, from the server. The electronic device may perform different operations according to a confidence score of a result of the ASR. Besides, it may be permissible to prepare other various embodiments speculated through the specification.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 14/680,444, filed on Apr. 7, 2015, which issues as U.S. Pat. No.9,640,183 on May 2, 2017 and claimed the benefit under 35 U.S.C. §119(e)of a U.S. Provisional application filed on Apr. 7, 2014 in the U.S.Patent and Trademark Office and assigned Ser. No. 61/976,142, and under35 U.S.C. §119(a) of a Korean patent application filed on Mar. 20, 2015in the Korean Intellectual Property Office and assigned Serial Number10-2015-0038857, the entire disclosure of each of which is herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates to a technology for executing speechinstructions to speech inputs of users by using a speech recognitionmodel, which is equipped in an electronic device, and a speechrecognition model available in a server.

BACKGROUND

In addition to traditional input methods of using a keyboard or a mouse,recent electronic devices may support input operations using speech. Forexample, electronic devices such as smart phones or tablet computers mayperform an operation of analyzing a user's speech that is input during aspecific function (e.g., S-Voice or Siri), converting the speech intotext, or executing a function corresponding to the speech. Someelectronic devices may normally remain in an always-on state for speechrecognition such that they may awake or be unlocked upon detection ofspeech, and perform functions of Internet surfing, telephoneconversations, SMS/e-mail readings, etc.

Although many technologies have been proposed for speech recognition, itis inevitable to encounter limitations to speech recognition inelectronic devices. For example, electronic devices may use speechrecognition models, which are embedded therein, for quick response tospeech recognition. However, the capacities of electronic devices arelimited which may cause a restriction in the number or kinds ofrecognizable speech inputs.

To obtain more accurate and reliable results for speech recognition,electronic devices may transmit speech inputs to a server to request theserver to recognize the speech inputs, provide results which are fedback from the server, or perform specific operations with reference tothe fed-back results. However, that manner could increase an amount ofcommunication traffic through the electronic devices and causerelatively slow response rates.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide a speech recognition method capable ofutilizing two or more different speech recognition capabilities ormodels to diminish inefficiency that may be encountered during theaforementioned diverse situations.

In accordance with an aspect of the present disclosure, an electronicdevice is provided. The electronic device includes a processorconfigured to perform an automatic speech recognition (ASR) on a speechinput by using a speech recognition model that is stored in a memory,and a communication module configured to provide the speech input to aserver and receive a speech instruction, which corresponds to the speechinput, from the server. The processor may further perform an operationcorresponding to a result of the ASR if a confidence score of the ASRresult is higher than a first threshold, and provide a feedback if aconfidence score of the ASR result is lower than a second threshold.

In accordance with another aspect of the present disclosure, a method ofexecuting speech recognition in an electronic device is provided. Themethod includes obtaining a speech input from a user, generating aspeech signal corresponding to the obtained speech, performing firstspeech recognition on at least a part of the speech signal, acquiringfirst operation information and a first confidence score, transmittingat least a part of the speech signal to a server for second speechrecognition, receiving second operation information, which correspondsto the transmitted signal, from the server, performing a functioncorresponding to the first operation information if the first confidencescore is higher than a first threshold value, providing a feedback ifthe first confidence score is lower than a second threshold value, andperforming a function corresponding to the second operation informationif the first confidence score is between the first threshold value andsecond threshold value.

In accordance with an aspect of the present disclosure, it may beadvantageous to increase a response rate and accuracy by executingspeech recognition by using a speech recognition model, which isprepared for an electronic device in itself, and using a result ofspeech recognition supplementary from a server which refers to a resultof the speech recognition by the speech recognition model.

Additionally, it may be permissible to compare results of speechrecognition between an electronic device and a server, and reflect aresult of the comparison to a speech recognition model or algorithmThen, it may be possible to continuously improve accuracy and responserate to permit repetitive speech recognition.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a block diagram illustrating an electronic device and aserver, which is connected to the electronic device through a network,according to an embodiment of the present disclosure;

FIG. 2 is a block diagram illustrating an electronic device and a serveraccording to embodiment of the present disclosure;

FIG. 3 is a flowchart showing a speech recognition method according toan embodiment of the present disclosure;

FIG. 4 is a flowchart showing a speech recognition method according toembodiment of the present disclosure;

FIG. 5 is a flowchart showing a method of updating a threshold accordingto an embodiment of the present disclosure;

FIG. 6 is a flowchart showing a method of updating a speech recognitionmodel according to an embodiment of the present disclosure;

FIG. 7 is a block diagram illustrating an electronic device in a networkenvironment according to an embodiment of the present disclosure; and

FIG. 8 is a block diagram illustrating an electronic device according toan embodiment of the present disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

As used herein, the terms “have,” “may have,” “include/comprise,” or“may include/comprise” indicate the existence of a corresponding feature(e.g., numerical values, functions, operations, or components/elements)but do not exclude the existence of other features.

As used herein, the terms “A or B,” “at least one of A or/and B,” or“one or more of A or/and B” may include all allowable combinations. Forinstance, the terms “at least one of A and B” or “at least one of A orB” may indicate (1) to include at least one A, (2) to include at leastone B, or (3) to include both at least one A and at least one B.

As used herein, the terms such as “1st,” “2nd,” “first,” “second,” andthe like used herein may refer to modifying various different elementsof various embodiments, but do not limit the elements. For instance,such terms do not limit the order and/or priority of the elements.Furthermore, such terms may be used to distinguish one element fromanother element. For instance, both “a first user device” and “a seconduser device” indicate a user device but indicate different user devicesfrom each other For example, a first component may be referred to as asecond component and vice versa without departing from the scope of thepresent disclosure.

As used herein, when one element (e.g., a first element) is referred toas being “operatively or communicatively connected with/to” or“connected with/to” another element (e.g., a second element), it shouldbe understood that the former may be directly coupled with the latter,or connected with the latter via an intervening element (e.g., a thirdelement). But, it will be understood that when one element is referredto as being “directly coupled” or “directly connected” with anotherelement, it means that there any intervening element is not existedbetween them.

In the description or claims, the term “configured to” (or “set to”) maybe changeable with other implicative meanings such as “suitable for,”“having the capacity of,” “designed to,” “adapted to,” “made to,” or“capable of,” and may not simply indicate “specifically designed to.”Alternatively, in some circumstances, a term such as “a deviceconfigured to” may indicate that the device “may do” something togetherwith other devices or components. For instance, the term “a processorconfigured to (or set to) perform A, B, and C” may indicate ageneric-purpose processor (e.g., CPU or application processor) capableof performing its relevant operations by executing one or more softwareor programs stored in an exclusive processor (e.g., embedded processor),which is prepared for the operations, or in a memory.

The terms in this specification are used to describe embodiments of thepresent disclosure and are not intended to limit the scope of thepresent disclosure. The terms of a singular form may include pluralforms unless otherwise specified. Unless otherwise defined, all theterms used herein, which include technical or scientific terms, may havethe same meaning that is generally understood by a person skilled in theart. It will be further understood that terms, which are defined in adictionary and commonly used, should also be interpreted as is customaryin the relevantly related art and not in an idealized or overly formalsense unless expressly so defined herein in various embodiments of thepresent disclosure. In some cases, terms even defined in thespecification may not be understood as excluding embodiments of thepresent disclosure.

Hereinafter, an electronic device according to various embodiments ofthe present disclosure will be described in more detail with referenceto the accompanied drawings. In the following description, the term“user” in various embodiments may refer to a person using an electronicdevice or a device using an electronic device (for example, anartificial intelligent electronic device).

FIG. 1 is a block diagram illustrating an electronic device and aserver, which is connected with the electronic device through a network,according to an embodiment of the present disclosure.

Referring to FIG. 1, the electronic device may include a configurationsuch as a user equipment (UE) 100. For example, the UE 100 may include amicrophone (microphone) 110, a controller 120, an Automatic SpeechRecognition (ASR) module 130, an ASR model 140, a transceiver 150, aspeaker 170, and a display 180. The configuration of the UE 100 shown inFIG. 1 is provided as an example. Thus, it may be modified by way ofvarious alternative embodiments of the present disclosure. For instance,the electronic device may include a configuration such as a UE 101 shownin FIG. 2, an electronic device 701 shown in FIG. 7, or an electronicdevice 800 shown in FIG. 8, or may be properly modified with thoseconfigurations. Hereinafter, various embodiments of the presentdisclosure will be described with reference to the UE 100.

The UE 100 may obtain a speech input through the microphone 110 from auser. For instance, if a user executes an application which is relevantto speech recognition, or if an operation of speech recognition is inactivation, the user's speech may be obtained by the microphone 110. Themicrophone 110 may include an analog-digital converter (ADC) to convertan analog signal into a digital signal. In some embodiments, an ADC, adigital-analog converter (DAC), a circuit processing diverse signals, ora pre-processing circuit may be included in the controller 120.

The controller 120 may provide a speech input, which is obtained by themicrophone 110, and an audio signal (or speech signal), which isgenerated from a speech input, to the ASR module 130 and the transceiver150. An audio signal provided to the ASR module 130 by the controller120 may be a signal which is pre-processed for speech recognition. Forinstance, an audio signal may be a signal which is noise-filtered orprocessed to be pertinent to human voice by an equalizer. Otherwise, asignal provided to the transceiver 150 by the controller 120 may be aspeech input itself. Different from the ASR module 130, the controller120 may transmit original speech data to the transceiver 150 to controla server 200 to work with a pertinent or more functional audio signalprocessing operation.

The controller 120 may control general operations of the UE 100. Forinstance, the controller 120 may control an operation for a speech inputfrom a user, an operation of speech recognition, and execution offunctions according to speech recognition.

The ASR module 130 may perform speech recognition on an audio signalwhich is provided from the controller 120. The ASR module 130 mayperform functions of isolated word recognition, connected wordrecognition, and large vocabulary recognition. The ASR performed by theASR module 130 may be implemented in a speaker-independent orspeaker-dependent type. The ASR module 130 may not be limited to asingle speech recognition engine, and may be formed of two or morespeech recognition engines. Additionally, if the ASR module 130 includesa plurality of speech recognition engines, each speech recognitionengine may be different one another in direction of recognition. Forinstance, one speech recognition engine may recognize wakeup speech,e.g., “Hi. Galaxy,” for activating an ASR function, while the otherspeech recognition engine may recognize command speech, e.g., “Read arecent e-mail.” The ASR module 130 may perform speech recognition withreference to the ASR model 140. Therefore, it may be permissible todetermine a range (e.g., kind or number) of speech input which isrecognizable by the ASR model 140. The aforementioned description aboutthe ASR module 130 may be applicable even to an ASR module 230 belongingto the server 200 which will be described later.

The ASR module 130 may convert a speech input into a text. The ASRmodule 130 may determine an operation or function which is to beperformed by the electronic device in response to a speech input.Additionally, the ASR module 130 may determine a confidence score orscore of a result of ASR.

The ASR model 140 may include grammar. Here, grammar may include varioustypes of grammar, which is statistically generated through a user'sinput or on the World Wide Web in addition to linguistic grammar. Invarious embodiments of the present disclosure, the ASR model 140 mayinclude an acoustic model, and a language model. Otherwise, the ASRmodel 140 may be a speech recognition model which is used for isolatedword recognition. In various embodiments of the present disclosure, theASR model 140 may include a recognition model for performing speechrecognition in a pertinent level in consideration of arithmetic andstorage capacities of the UE 100. For instance, the grammar may,regardless of linguistic grammar, include grammar for a designatedinstruction structure. For example, “call [user name]” corresponds togrammar for sending a call to a user of [user name], and may be includedin the ASR model 140.

The transceiver 150 may transmit a speech signal, which is provided fromthe controller 120, to the server 200 by way of a network 10.Additionally, the transceiver 150 may receive a result of speechrecognition, which corresponds to the transmitted speech signal, fromthe server 200.

The speaker 170 and the display 110 may be used for interacting with auser's input. For instance, if a speech input is provided from a userthrough the microphone 110, a result of speech recognition may beexpressed in the display 180 and output through the speaker 170.Needless to say, the speaker 170 and the display 180 may also performgeneral functions of outputting sound and a screen in the UE 100.

The server 200 may include a configuration for performing speechrecognition with a speech input which is provided from the UE 100 by wayof the network 20. Then, partial elements of the server 200 maycorrespond to the UE 100. For instance, the server 200 may include atransceiver 210, a controller 220, the ASR module 230, and an ASR model240. Additionally, the server 200 may further include an ASR modelconverter 250, or a natural language processing (NLP) unit 260.

The controller 220 may control functional modules for performing speechrecognition in the server 200. For instance, the controller 220 may becoupled with the ASR module 230 and/or the NLP 260. Additionally, thecontroller 220 may cooperate with the UE 100 to perform a functionrelevant to recognition model update. Additionally, the controller 220may perform a pre-processing operation with a speech signal, which istransmitted by way of the network 10, and provide a pre-processed speechsignal to the ASR module 230. This pre-processing operation may bedifferent from the pre-processing operation, which is performed in theUE 100, in type or effect. In some embodiments, the controller 220 ofthe server 200 may be referred to as an orchestrator.

The ASR module 230 may perform speech recognition with a speech signalwhich is provided from the controller 220. The above descriptionregarding the ASR module 130 may be at least partially applied to theASR module 230. While the ASR module 230 for the server 200 and the ASRmodule 130 for the UE 100 are described as performing partially similarfunctions, they may be different each other in functional boundary oralgorithm. The ASR module 230 may perform speech recognition withreference to the ASR model 130, and then generate a result that isdifferent from a speech recognition result of the ASR module 130 of theUE 100. In more detail, the server 200 may generate a recognition resultthrough the ASR module 230 and the NLP 260 by means of speechrecognition, natural language understanding (NLU), Dialog Management(DM), or a combination thereof, while the UE 100 may generate arecognition result through the ASR module 130. For instance, firstoperation information and a first confidence score may be determinedafter an ASR operation of the ASR module 130, and second operationinformation and a second confidence score may be determined after an ASRoperation of the ASR module 230. In some embodiments, results from theASR modules 130 and 230 may be identical to each other, or may bedifferent in at least one part. For instance, the first operationinformation corresponds to the second operation information, but thefirst confidence score may be higher than the second confidence score.In various embodiments of the present disclosure, an ASR operationperformed by the ASR module 130 of the UE 100 may be referred to as“first speech recognition” while an ASR operation performed by the ASRmodule 230 of the server 200 may be referred to as “second speechrecognition.”

In various embodiments of the present disclosure, if the first speechrecognition performed by the ASR module 130 is different from the secondspeech recognition performed by the ASR module 230 in algorithm or inusage model, the ASR model converter 250 may be included in the server200 to change a model type between them.

Additionally, the server 200 may include the NLP 260 for sensing auser's intention and determining a function, which is to be performed,with reference to a recognition result of the ASR module 230. The NLP260 may perform natural word understanding, which mechanically analyzesan effect of words spoken by humans and then makes the words intocomputer-recognizable words, or reversely, a natural word processingthat expresses human-understandable words from the computer-recognizablewords.

FIG. 2 is a block diagram illustrating an electronic device and a serveraccording to embodiment of the present disclosure.

Referring to FIG. 2, an electronic device is exemplified in aconfiguration different from that of FIG. 1. However, a speechrecognition method disclosed in this specification may also be performedby an electronic device/user equipment shown in FIG. 1, FIG. 2, FIG. 7,or FIG. 8, some of which will be described below, by another devicewhich can be modifiable or variable from the electronic device/userequipment.

Referring to FIG. 2, a UE 101 may include a processor 121 and a memory141. The processor 121 may include an ASR engine 131 for performingspeech recognition. The memory 141 may store an ASR model 143 which isused by the ASR engine 131 to perform speech recognition. For instance,considering functions performed by elements of the configuration, it canbe seen that the processor 121, the ASR engine 131, and the ASR model(or the memory 141) of FIG. 2 may correspond respectively with thecontroller 120, the ASR model 130, and the ASR model 140 of FIG. 1.Thus, a duplicative description will not be further offered hereinafter.

The UE 101 may obtain a speech input from a user by using a speechrecognition (i.e., acquisition) module 111 (e.g., the microphone 110).The processor 121 may perform an ASR operation to the obtained speechinput by means of the ASR model 143 which is stored in the memory 141.Additionally, the UE 101 may provide a speech input to the server 200 byway of a communication module 151, and receive a speech instruction(e.g., a second operation information), which corresponds to an speechinput, from the server 200. The UE 101 may output a speech recognitionresult, which is acquisitive by the ASR engine 131 and the server 200,through a display 181 (or speaker).

Hereinafter, diverse speech recognition methods will be described on aUE 100 in conjunction with FIGS. 3 to 6.

FIG. 3 is a flowchart showing a speech recognition method according toan embodiment of the present disclosure.

Referring to FIG. 3, the UE 100 may obtain a user's speech input byusing a speech acquisition module such as microphone in operation 301.This operation may be performed in the state that the user executes aspecific function or application which is relevant to speechrecognition. But in some embodiments, speech recognition of the UE 100may be conditioned in an always-on state (e.g., the microphone isnormally turned on), for which operation 301 may be normally active to auser's speech. Otherwise, an ASR operation may be conditioned to be anactive state by different speech recognition engines in response to aspecific speech input (e.g., “Hi, Galaxy”), as aforementioned, and thenperformed with subsequently input speech recognition information.

In operation 303, the UE 100 may transmit a speech signal (or at least apart of a speech signal) to the server 200. In an internal view for thedevice, a speech signal (e.g., an audio signal made by a speech input isconverted into a (digital) speech signal and by pre-processing thespeech signal) may be provided to the ASR module 130 by a processor(e.g., the controller 120). In other words, in operation 303, the UE 100may provide a speech signal, which is regarded as a target ofrecognition, to an ASR module which is placed in and out of a devicecapable of performing speech recognition. The UE 100 may utilize all ofspeech recognition capabilities that are prepared in itself and theserver 200.

In operation 305, the UE 100 may perform speech recognition by itself.This speech recognition may be referred to as “ASR1.” For instance, theASR module 130 may perform speech recognition with a speech input byusing the ASR model 140. For instance, the ASR model 140 may performASR1 with at least a part of a speech signal. After performing ASR1, aresult of speech recognition may be obtained. For instance, if a userprovides a speech input such as “tomorrow weather,” the UE 100 maydetermine operation information such as “weather application, tomorrowweather output” by using a function of speech recognition to the speechinput. Additionally, a result of speech recognition may include aconfidence score of operation information. For instance, although theASR module 130 may determine a confidence score of 95% if a user'sspeech is analyzed as clearly indicating “tomorrow weather,” the ASRmodule 130 may also give a confidence score of 60% to a determinedoperation information even if a user's speech is analyzed as being vagueto indicate “everyday weather” or “tomorrow weather.”

In operation 307, a processor may determine whether a confidence scoreis higher than a threshold. For instance, if a confidence score ofoperation information determined by the ASR module 130 is higher than apredetermined level (e.g., 80%), the UE 100 may perform ASR1, i.e. anoperation corresponding to a speech instruction recognized by a speechrecognition function of the UE 100 in itself, in operation 309. Thisoperation may be performed with at least one function that ispracticable by the processor, at least one application, or at least oneof inputs based on an execution result of ASR operation.

Operation 309 may be performed before a speech recognition result isreceived from the server 200 (e.g., before operation 315). In otherwords, if speech recognition self-performed by the UE 100 results in asufficient confidence score to recognize a speech instruction, the UE100 may directly perform a corresponding operation, without waiting foran additional result of speech recognition which is received from theserver 200, to provide a quick response time to a user's speech input.

If a confidence score is less than the threshold in operation 307, theUE 100 may be maintained in a standby state until a speech recognitionresult is received from the server 200. During the standby state, the UE100 may display a suitable message, icon, or image to inform a user thatspeech recognition is operating to the speech input.

In operation 311, speech recognition by the server 200 may be performedwith a speech signal which is transmitted to the server 200 in operation303. This speech recognition may be referred to as “ASR2.” Additionally,an NLP may be performed in operation 313. For instance, an NLP may beperformed with a speech input or a recognition result of ASR2 by usingthe NLP 260. In some embodiments, this operation may be performed byselection of the user.

In operation 315, if speech recognition results (e.g., a secondoperation information and a second confidence score) of ASR1, ASR2, orNLP are received from the server 200, operation 317 may include anoperation corresponding to a speech instruction (e.g., second operationinformation) by way of ASR2. Since operation 317 needs to allow anadditional time for transmitting a speech signal at operation 303 andacquiring a speech recognition result at operation 315, it takes alonger time than operation 309. However, it may be possible to perform aspeech recognition operation with higher confidence score and accuracyeven in comparison with a case of speech recognition that operation 317is incapable of self-processing or capable of self-processing butresulting in a low confidence score.

FIG. 4 is a flowchart showing a speech recognition method according toan embodiment of the present disclosure.

Referring to FIG. 4, as speech recognition operation 401, speech signaltransmit operation 403, ASR1 operation 405, ASR2 operation 415, and NLPoperation 417 correspond respectively to the aforementioned operations301, 303, 305, 311, and 313, those operations will not be furtherdescribed below.

A speech recognition method described in conjunction with FIG. 4 may beperformed by referring to two thresholds. Based on a first threshold anda second threshold that is lower than the first threshold in confidencescore, different operations (e.g., operations 409, 413, and 421,respectively) may be performed respectively for the cases that aconfidence score of ASR1 result from operation 405 is: (1) higher thanthe second threshold; (2) lower than the second threshold; and (3)between the first and second thresholds.

If a confidence score is determined as being higher than the firstthreshold in operation 407, the UE 100 may perform an operationcorresponding to an ASR1 result in operation 409. If a confidence scoreis determined as being lower than the first threshold in operation 407,the process may go to determine whether the confidence score is lowerthan the second threshold in operation 411.

In operation 411, if a confidence score is lower than the secondthreshold, the UE 100 may provide a feedback for the confidence score.This feedback may include a message or an audio output which indicatesthat a user's speech input was abnormally recognized, or normallyrecognized but in lack of confidence. For instance, the UE 100 maydisplay or output a guide message, such as “Your speech is notrecognized, Please speak again,” through a screen or a speaker.Otherwise, the UE 100 may confirm accuracy of a low-confidentrecognition result by guiding a user to a relatively easy-recognizablespeech input (e.g., “Yes,” “Not,” “No,” “Impossible,” “Never,” and soon) by way of a feedback such as “Did you speak XXX?”

Once a feedback is provided in operation 413, operation 421 may not beperformed even if a speech recognition result is obtained in operation409 along a lapse of time later. This is because a feedback may cause anew speech input from a user and then it may be unreasonable to performan operation with the previous speech input. But in some embodiments,operation 421 may be performed after operation 413 if there is noadditional input from a user for a predetermined time, despite afeedback of operation 413, and if a speech recognition result (e.g., asecond operation information and a second confidence score), which isreceived from the server 200 in operation 419, satisfies a predeterminedcondition (e.g., the second confidence score is higher than the firstthreshold or a certain third threshold).

In operation 411, if a confidence score obtained from operation 405 ishigher than the second threshold (i.e. the confidence score ranksbetween the first and second thresholds), the UE 100 may receive aspeech recognition result from the server 200 in operation 419. Inoperation 421, the UE 100 may perform an operation which corresponds toa speech instruction (second operation information) by way of ASR2.

In the embodiment shown in FIG. 4, it may be permissible todifferentiate a confidence score, which results from speech recognitionby the UE 100, into a usable level with reference to usable and unusablelevels and an ASR result of the server 200, and then enable an operationthat is pertinent to the differentiated confidence score. Especially, ifa confidence score is excessively low, the UE 100 may provide afeedback, regardless of reception of a result from the server 200, toguide a user to a speech re-input, and may thereby prevent a message,such as “not recognized,” from being provided to a user after a longtime from a response standby time.

FIG. 5 is a flowchart showing a method of updating a threshold accordingto an embodiment of the present disclosure.

Referring to FIG. 5, as speech acquisition operation 501, speech signaltransmit operation 503, ASR1 operation 505, ASR2 operation 511, and NLPoperation 513 correspond respectively to the aforementioned operations301, 303, 305, 311, and 313, those operations will not be furtherdescribed below.

If a confidence score of a result from ASR1 is determined to be largerthan a threshold (e.g., a first threshold) in operation 507, the processmay go to operation 509 to perform an operation corresponding to aspeech instruction (e.g., first operation information) by way of ASR1.If a confidence score of an ASR1 result is determined as being lowerthan the threshold, an operation subsequent to operation 315 of FIG. 3or operation 411 of FIG. 4 may be performed.

In the embodiment of FIG. 5, even after operation 509, the process maynot be terminated but may continue to perform operations 515 through517. In operation 515, the UE 100 may receive a speech recognitionresult from the server 200. For instance, the UE 100 may obtain secondoperation information and a second confidence score, which result fromASR2, to a speech signal transmitted during operation 503.

In operation 517, the UE 100 may compare ASR1 with ASR2 in recognitionresult. For instance, the UE 100 may determine whether recognitionresults from ASR1 and ASR2 are identical to or different from eachother. For instance, if ASR1 recognizes speech as “Tomorrow weather” andASR2 recognizes speech as “Tomorrow weather?,” both may includeoperation information such as “Output weather application, Outputtomorrow weather.” In this case, it can be understood that suchrecognition results of ASR1 and ASR2 may correspond each other.Otherwise, if different operations are performed after speechrecognition, the two (or more) speech recognition results may bedetermined as none-corresponding each other.

In operation 519, the UE 100 may change a threshold by comparing aresult of ASR1 (self-operation of speech recognition in the UE 100) witha speech instruction which is received from the server 200. Forinstance, the UE 100 may decrease the first threshold if the firstoperation information and the second operation information are identicaleach other or include a speech instruction corresponding thereto. Forinstance, for a certain speech input, a general method is designed tocontrol a speech recognition result by itself from the UE 100 not to beadopted, without waiting for a response from the server 200, until aconfidence score reaches 80%, whereas this method may be designed tocontrol a confidence score higher even than 70% to enable a speechrecognition result by itself from the UE 100 to be adopted by way ofthreshold update. Threshold update may be resumed whenever a user playsthe speech recognition function and thus it may shorten a response timebecause speech recognition frequently operating by a user is set to havea lower threshold.

In the meantime, if ASR1 is different from ASR2 in result, the thresholdmay increase. In some embodiments, an operating of updating a thresholdmay occur after a predetermined condition is accumulated as many as thepredetermined number of times. For instance, for a certain speech input,if results from ASR1 and ASR2 agree with each other in more than fivetimes, a threshold may be updated lower.

FIG. 6 is a flowchart showing a method of updating a speech recognitionmodel according to an embodiment of the present disclosure.

Referring to FIG. 6, as speech recognition operation 601, speech signaltransmit operation 603, ASR1 operation 605, ASR2 operation 611, and NLPoperation 613 correspond respectively to the aforementioned operations301, 303, 305, 311, and 313, those operations will not be furtherdescribed below.

In operation 607, if a confidence score of a result of ASR1 isdetermined as being greater than a threshold (e.g., a first threshold),an operation subsequent to operation 309 of FIG. 3, operation 409 ofFIG. 4, and operation 509 of FIG. 5 may be performed.

If a confidence score of a result of ASR1 is determined as being lowerthan the threshold in operation 607, the UE 100 may receive a speechrecognition result from the server 200 in operation 609 and in operation615, perform an operation corresponding to a speech instruction by wayof ASR2. Operations 609 and 615 may correspond to operations 315 and 317of FIG. 3, or operations 419 and 421 of FIG. 4.

In operation 617, the UE 100 may compare ASR1 with ASR2 in speechrecognition result. Operation 617 may correspond to operation 517 ofFIG. 5.

In operation 619, the UE 100 may update its own speech recognition model(e.g., the ASR model 140) with reference to a comparison result fromoperation 617. For instance, the UE 100 may add a speech recognitionresult (e.g., second operation information, or the second operationinformation and a second confidence score) of ASR2, which is generatedin response to a speech input, to the speech recognition model. Forinstance, if the first operation information does not correspond to thesecond operation information, the UE 100 may add the second operationinformation (and the second confidence score) to a speech recognitionmodel, which is used for the first speech recognition, with reference tothe first and second confidence scores (e.g., if the second confidencescore is higher than the first confidence score). Similar to theembodiment shown in FIG. 5, an operation of updating a speechrecognition model may occur after a predetermined condition isaccumulated a predetermined number of times.

FIG. 7 is a block diagram illustrating an electronic device in a networkenvironment according to an embodiment of the present disclosure.

Referring to FIG. 7, an electronic device 710 may be situated in anetwork environment 700 in accordance with various embodiments. Theelectronic device 701 may include a bus 710, a processor 720, a memory730, an input/output (I/O) interface 750, a display 760, and acommunication interface 770. In some embodiments, the electronic device701 may be organized without at least one of the elements, or comprisedof another additional element.

The bus 710 may include, for example, a circuit to interconnect theelements 710˜770 and help communication (e.g., control messages and/ordata) between the elements.

The processor 720 may include one or more of central processing unit(CPU), application processor (AP), or communication processor (CP). Theprocessor 720 may perform for example an arithmetic operation or dataprocessing to control and/or communicate at least one of other elements.

The memory 730 may include a volatile and/or nonvolatile memory. Thememory 730 may store for example instructions or data that are involvedin at least one of other elements. According to an embodiment, thememory 730 may store a software and/or program 740. The program 740 mayinclude for example a kernel 741, a middleware 743, an applicationprogramming interface (API) 745, and/or application programs (or“application”) 747. At least one of the kernel 741, the middleware 743,or the API 745 may be called “Operating System (OS).”

The kernel 741 may control or manage, for example, system resources(e.g., the bus 710, the processor 720, or the memory 730) which are usedfor executing an operation or function implemented embodied in otherprograms (e.g., the middleware 743, the API 745, or the applicationprograms 747). Additionally, the kernel 741 may provide an interfacecapable of controlling or managing system resources by approachingindividual elements of the electronic device 701 in the middleware 743,the API 745, or the application programs 747.

The middleware 743 may intermediate, for example, to manage the API 745or the application programs 747 to communicate data with the kernel 741.

Additionally, the middleware 743 may process one or more work requests,which are received from the application programs 747, in priority. Forinstance, the middleware 743 may allow at least one of the applicationprograms 747 to have priority capable of using a system resource (e.g.,the bus 710, the processor 720, or the memory 730) of the electronicdevice 701. For instance, the middleware 743 may perform scheduling orload balancing to the at least one or more work requests by processingthe at least one or more work requests in accordance with the priorityallowed for the at least one of the application programs 747.

The API 745, as an interface necessary for the application 747 tocontrol a function that is provided from the kernel 741 or themiddleware 743, may be include foe example at least one interface orfunction (e.g., instruction) for file control, window control, orcharacter control.

The I/O interface 750 may act, for example, as an interface capable oftransmitting instructions or data, which are input from a user oranother external system, to another element (or other elements) of theelectronic device 701. Additionally, the I/O interface 750 may outputinstructions or data, which are received from another element (or otherelements) of the electronic device 701, to a user or another externalsystem.

The display 760 may include, for example, a Liquid Crystal Display(LCD), a light emission diode (LED), an organic light emission Diode(OLED) display, or a micron electro-mechanical system (MEMS) display, oran electronic paper display. The display 760 may express, for example,diverse contents (e.g., text, image, video, icon, or symbol) to a user.The display 760 may include a touch screen and for example, receive aninput by touch, gesture, approach, or hovering which is made with a partof a user's body or an electronic pen.

The communication interface 770 may set, for example, communicationbetween the electronic device 710 and an external device (e.g., a firstexternal electronic device 702, a second external electronic device 704,or a server 706). For instance, the communication interface 770 maycommunicate with the external device (e.g., the second externalelectronic device 704 or the server 706) in connection with a network762 through wireless or wired communication.

Wireless communication may adopt at least one of LTE, LTE-A, CDMA,WCDMA, UMTS, WiBro, and GSM for cellular communication protocol.Additionally, wireless may include for example a local areacommunication 764. The local area communication 764 may include, forexample, at least one of Wi-Fi, Bluetooth, near field communication(NFC), or global positioning system (GPS). Wired communication mayinclude, for example, at least one of universal serial bus (USB), highdefinition multimedia Interface (HDMI), recommended standard 832(RS-232), and plain old telephone server (POTS). The network 762 mayinclude a communication network, for example, at least one of a computernetwork (e.g., LAN or WAN), the Internet, and a telephone network.

The first and second external electronic devices 702 and 704 may be samewith or different from the electronic device 701. According to anembodiment, the server 706 may include one or more groups of servers. Invarious embodiments of the present disclosure, all or a part ofoperations performed in the electronic device 701 may be also performedin another one or a plurality of electronic devices (e.g., theelectronic devices 702 and 704, or the server 706). According to anembodiment, if there is a need to perform some function or service byautomation or request, the electronic device 701 may request anotherdevice (e.g., the electronic device 702 or 704, or the server 706) toperform such function or service, instead of executing the function orservice in itself, or request such other devices to perform the functionor service to perform in addition to its self-execution. Such anotherelectronic device (e.g., the electronic device 702 or 704, or the server706) may perform the requested function or service, and then transmit aresult thereof to the electronic device 701. The electronic device 701may process the received result directly or additionally to provide therequested function or service. For this operation, for example, it maybe allowable to employ cloud computing, dispersion computing, orclient-server computing technology.

FIG. 8 is a block diagram illustrating an electronic device according toan embodiment of the present disclosure.

Referring to FIG. 8, an electronic device 800 may include, for example,all or a part of elements of the electronic device 701 shown in FIG. 7.Referring to FIG. 8, the electronic device 800 may include at least oneof one or more Application Processors (AP) 810, a communication module820, a subscriber identification module (e.g., SIM card) 824, a memory830, a sensor module 840, an input unit 850, a display 860, an interface870, an audio module 880, a camera module 891, a power management module895, a battery 896, an indicator 897, or a motor 898.

The processor (AP) 810 may drive an OS or an application to control aplurality of hardware or software elements connected to the processor810 and may process and compute a variety of data including multimediadata. The processor 810 may be implemented with a system-on-chip (SoC),for example. According to an embodiment, the AP 810 may further includea graphic processing unit (GPU) and/or an image signal processor. Theprocessor 810 may even include at least a part of the elements shown inFIG. 8. The processor 810 may process instructions or data, which arereceived from at least one of other elements (e.g., a nonvolatilememory), and then store diverse data into such a nonvolatile memory.

The communication module 820 may have a configuration same with orsimilar to the communication interface 770 of FIG. 7. The communicationmodule 820 may include a cellular module 821, a WiFi module 823, aBluetooth (BT) module 825, a GPS module 827, an NFC module 828, and aradio frequency (RF) module 829.

The cellular module 821 may provide voice call, video call, a characterservice, or an Internet service through a communication network.According to an embodiment, the cellular module 821 may performdiscrimination and authentication of an electronic device within acommunication network using a subscriber identification module (e.g., aSIM card) 824. According to an embodiment, the cellular module 821 mayperform at least a portion of functions that the AP 810 provides.According to an embodiment, the cellular module 821 may include a CP.

Each of the WiFi module 823, the Bluetooth module 825, the GPS module827, and the NFC module 828 may include a processor for processing dataexchanged through a corresponding module, for example. In someembodiments, at least a part (e.g., two or more elements) of thecellular module 821, the WiFi module 823, the Bluetooth module 825, theGPS module 827, and the NFC module 828 may be included within oneintegrated circuit (IC) or an IC package.

The RF module 829 may transmit and receive, for example, communicationsignals (e.g., RF signals). The RF module 829 may include a transceiver,a power amplifier module (PAM), a frequency filter, a low noiseamplifier (LNA), or an antenna. According to another embodiment, atleast one of the cellular module 821, the WiFi module 823, the Bluetoothmodule 825, the GPS module 827, and the NFC module 828 may transmit andreceive an RF signal through a separate RF module.

The SIM card 824 may include, for example, a card, which has asubscriber identification module, and/or an embedded SIM, and includeunique identifying information (e.g., Integrated Circuit Card Identifier(ICCID)) or subscriber information (e.g., Integrated Mobile SubscriberIdentify (IMSI)).

The memory 830 (e.g., the memory 730) may include, for example, anembedded memory 832 or an external memory 834. For example, the embeddedmemory 832 may include at least one of a volatile memory (e.g., adynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM(SDRAM), etc.), a nonvolatile memory (e.g., a one-time programmable ROM(OTPROM), a programmable ROM (PROM), an erasable and programmable ROM(EPROM), an electrically erasable and programmable ROM (EEPROM), a maskROM, a flash ROM, a NAND flash memory, a NOR flash memory, etc.), a harddrive, or solid state drive (SSD).

The external memory 834 may further include a flash drive, for example,a compact flash (CF), a secure digital (SD), a micro-secure Digital(SD), a mini-SD, an extreme Digital (xD), or a memory stick. Theexternal memory 834 may be functionally connected with the electronicdevice 800 through various interfaces.

The sensor module 840 may measure, for example, a physical quantity, ordetect an operation state of the electronic device 800, to convert themeasured or detected information to an electric signal. The sensormodule 840 may include at least one of a gesture sensor 840A, a gyrosensor 840B, a pressure sensor 840C, a magnetic sensor 840D, anacceleration sensor 840E, a grip sensor 840F, a proximity sensor 840G, acolor sensor 840H (e.g., RGB sensor), a living body sensor 840I, atemperature/humidity sensor 840J, an illuminance sensor 840K, or an UVsensor 840M. Additionally or generally, though not shown, the sensormodule 840 may further include an E-nose sensor, an electromyographysensor (EMG) sensor, an electroencephalogram (EEG) sensor, anElectroCardioGram (ECG) sensor, a photoplethysmography (PPG) sensor, aninfrared (IR) sensor, an iris sensor, or a fingerprint sensor, forexample. The sensor module 840 may further include a control circuit forcontrolling at least one or more sensors included therein. In someembodiments, the electronic device 800 may further include a processor,which is configured to control the sensor module 840, as a part oradditional element, thus enabling to control the sensor module 840 whilethe processor 810 is in a sleep state.

The input unit 850 may include, for example, a touch panel 852, a(digital) pen sensor 854, a key 856, or an ultrasonic input unit 858.The touch panel 852 may recognize, for example, a touch input using atleast one of a capacitive type, a resistive type, an infrared type, oran ultrasonic wave type. Additionally, the touch panel 852 may furtherinclude a control circuit. The touch panel 852 may further include atactile layer to provide a tactile reaction for a user.

The (digital) pen sensor 854 may be a part of the touch panel 852, or aseparate sheet for recognition. The key 856, for example, may include aphysical button, an optical key, or a keypad. The ultrasonic input unit858 may allow the electronic device 800 to detect a sound wave using amicrophone (e.g., a microphone 888), and determine data through an inputtool generating an ultrasonic signal.

The display 860 (e.g., the display 760) may include a panel 862, ahologram device 864, or a projector 866. The panel 862 may include thesame or similar configuration with the display 760 of FIG. 7. The panel862, for example, may be implemented to be flexible, transparent, orwearable. The panel 862 and the touch panel 852 may be implemented withone module. The hologram device 864 may show a three-dimensional imagein a space using interference of light. The projector 866 may projectlight onto a screen to display an image. The screen, for example, may bepositioned in the inside or outside of the electronic device 800.According to an embodiment, the display 860 may further include acontrol circuit for controlling the panel 862, the hologram device 864,or the projector 866.

The interface 870, for example, may include a high-definition Multimediainterface (HDMI) 872, a USB 874, an optical interface 876, or a D-sub(D-subminiature) 878. The interface 870 may include, for example, thecommunication interface 770 shown in FIG. 7. The interface 870, forexample, may include a mobile high definition Link (MHL) interface, anSD card/multi-media cared (MMC) interface, or an Infrared DataAssociation (IrDA) standard interface.

The audio module 880 may convert a sound and an electric signal in dualdirections. At least one element of the audio module 880 may include,for example, the I/O interface 750 shown in FIG. 7. The audio module880, for example, may process sound information that is input or outputthrough a speaker 882, a receiver 884, an earphone 886, or themicrophone 888.

The camera module 891 may be a unit which is capable of taking a stillpicture and a moving picture. According to an embodiment, the cameramodule 891 may include one or more image sensors (e.g., a front sensoror a rear sensor), a lens, an image signal processor (ISP), or a flash(e.g., an LED or a xenon lamp).

The power management module 895 may manage, for example, power of theelectronic device 800. The power management module 895 may include, forexample, a power management integrated Circuit (PMIC) a charger IC, or abattery or fuel gauge. The PMIC may operate in wired and/or wirelesscharging mode. A wireless charging mode may include, for example,diverse types of magnetic resonance, magnetic induction, orelectromagnetic wave. For the wireless charging, an additional circuit,such as a coil loop circuit, a resonance circuit, or a rectifier, may befurther included therein. The battery gauge, for example, may measure aremnant of the battery 896, a voltage, a current, or a temperature, forexample during charging. The battery 896 may measure, for example, aresidual capacity, a voltage on charge, a current, or temperaturethereof. The battery 896 may include, for example, a rechargeablebattery and/or a solar battery.

The indicator 897 may display the following specific state of theelectronic device 800 or a part (e.g., the AP 9810) thereof: a bootingstate, a message state, or a charging state. The motor 898 may convertan electric signal into mechanical vibration and generate a vibration orhaptic effect. Although not shown, the electronic device 800 may includea processing unit (e.g., a GPU) for supporting a mobile TV. Theprocessing unit for supporting the mobile TV, for example, may processmedia data that is based on the standard of digital multimediabroadcasting (DMB), digital video broadcasting (DVB), or media flow(MediaFlo™).

Each of the above components (or elements) of the electronic deviceaccording to an embodiment of the present disclosure may be implementedusing one or more components, and a name of a relevant component mayvary with on the kind of the electronic device. The electronic deviceaccording to various embodiments of the present disclosure may includeat least one of the above components. Also, a part of the components maybe omitted, or additional other components may be further included.Also, some of the components of the electronic device according to thepresent disclosure may be combined to form one entity, thereby making itpossible to perform the functions of the relevant componentssubstantially the same as before the combination.

The term “module” used for the present disclosure, for example, may meana unit including one of hardware, software, and firmware or acombination of two or more thereof. A “module,” for example, may beinterchangeably used with terminologies such as a unit, logic, a logicalblock, a component, a circuit, etc. The “module” may be a minimum unitof a component integrally configured or a part thereof. The “module” maybe a minimum unit performing one or more functions or a portion thereof.The “module” may be implemented mechanically or electronically. Forexample, the “module” according to the present disclosure may include atleast one of an application-specific integrated circuit (ASIC) chipperforming certain operations, a field-programmable gate arrays (FPGAs),or a programmable-logic device, known or to be developed in the future.

At least a part of an apparatus (e.g., modules or functions thereof) ormethod (e.g., operations or operations) according to various embodimentsof the present disclosure, for example, may be implemented byinstructions stored in a computer-readable storage medium in the form ofprogrammable module.

For example, the storage medium may store instructions enabling, duringexecution, an operation (or operation) of allowing a processor of anelectronic device to obtain a speech input and then generate a speechsignal, an operation of performing first speech recognition to at leasta part of the speech signal to obtain first operation information and afirst confidence score, an operation of transmitting at least a part ofthe speech signal to a server for the second speech recognition, and anoperation of receiving second operation information to the signaltransmitted from the server, and functions of (1) corresponding to thefirst operation information if the first confidence score is higher thana first threshold, (2) providing a feedback to the first confidencescore if the first confidence score is lower than a second threshold,and (3) corresponding to the second operation information if the firstconfidence score is between the first and second thresholds.

A module or programming module according to various embodiments of thepresent disclosure may include at least one of the above elements, or apart of the above elements may be omitted, or additional other elementsmay be further included. Operations performed by a module, a programmingmodule, or other elements according to an embodiment of the presentdisclosure may be performed sequentially, in parallel, repeatedly, or ina heuristic method. Also, a portion of operations may be performed indifferent sequences, omitted, or other operations may be added.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. An electronic device comprising: a processorconfigured to perform automatic speech recognition (ASR) on a speechinput by using a speech recognition model that is stored in a memory;and a communication module configured to provide the speech input to aserver and receive a speech instruction, which corresponds to the speechinput, from the server, wherein the processor is further configured to:perform an operation corresponding to a result of the ASR if aconfidence score of the result of the ASR is higher than a firstthreshold value, and provide a feedback if the confidence score of theresult of the ASR is lower than a second threshold value.